Multiverse Meta-Analysis with metaMultiverse

Author

Constantin Yves Plessen

Published

March 31, 2025

Introduction to Multiverse Meta-Analysis

Multiverse meta-analysis is an approach that addresses the analytical flexibility inherent in meta-analysis by exploring multiple reasonable analytical choices simultaneously. This package provides tools for conducting multiverse pairwise meta-analyses with standardized mean differences.

What is a multiverse analysis?

A multiverse analysis considers all reasonable analytical choices simultaneously, rather than committing to a single analytical path. This approach acknowledges the potential impact of researcher decisions on results and provides a more complete picture of possible outcomes.

Package Philosophy

The metaMultiverse package is designed to:

  1. Enable exploration of different meta-analytic choices (“which” factors and “how” factors)
  2. Visualize the impact of these choices on effect size estimates
  3. Assess the robustness of conclusions across the multiverse of possible analyses

Installation

You can install the development version of metaMultiverse from GitHub:

Code 
# install.packages("devtools")
devtools::install_github("cyplessen/metaMultiverse")

Workflow Overview

The typical workflow for a multiverse meta-analysis using this package consists of four main steps:

Prepare and validate your dataset to ensure it meets the requirements for multiverse analysis.

Define the analytical choices to explore, including “which” factors (data subsets) and “how” factors (meta-analytic methods).

Apply specifications to create a multiverse of meta-analyses, exploring all reasonable analytical paths.

Explore the results through specification curves and other visualizations to assess robustness.

This tutorial walks through each step with a practical example.

Example Dataset

For this tutorial, we’ll use a dataset of digital depression interventions:

Code 
# Load required packages
library(metaMultiverse)
library(tidyr)
library(dplyr) # data wrangling
library(ggplot2) # data visualization
library(plotly) # interactive data visualization
library(kableExtra) # data formatting

# Load example dataset
data("data_digDep") 

# Examine the structure
data_digDep %>%
  kbl() %>%
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed", "responsive"),
    full_width = FALSE,
    position = "center"
  ) %>%
  scroll_box(width = "100%", height = "300px")
study es_id yi vi sei wf_1 wf_2 wf_3 wf_4 wf_5 wf_6 wf_7 wf_8 condition_arm1 condition_arm2
Addington, 2019 1 0.2228951 0.1057582 0.3252049 website minimal to no support adul not-cbt-based wl cut some concerns post other psy wl
Addington, 2019 2 0.2734972 0.0981878 0.3133493 website minimal to no support adul not-cbt-based other ctr cut some concerns post other psy other ctr
Addington, 2019 3 0.3206356 0.1064319 0.3262391 website minimal to no support adul not-cbt-based wl cut some concerns post other psy wl
Addington, 2019 4 0.2134963 0.0978351 0.3127861 website minimal to no support adul not-cbt-based other ctr cut some concerns post other psy other ctr
Alavi, 2016 5 1.2933851 0.0521600 0.2283857 website guided adul cbt-based wl cut high risk post cbt wl
Alavi, 2016 6 1.3747302 0.0533464 0.2309685 website guided adul cbt-based wl cut high risk follow-up cbt wl
Andersson, 2005 7 0.9544523 0.0536431 0.2316098 website guided adul cbt-based other ctr cut some concerns post cbt other ctr
Andersson, 2005 8 0.7899661 0.0519243 0.2278690 website guided adul cbt-based other ctr cut some concerns post cbt other ctr
Araya, 2021a 9 0.2965846 0.0069548 0.0833954 mobile guided med cbt-based cau cut some concerns post bat cau
Araya, 2021a 10 0.1294997 0.0065934 0.0812000 mobile guided med cbt-based cau cut some concerns follow-up bat cau
Araya, 2021b 11 0.4205505 0.0125420 0.1119913 mobile guided med cbt-based cau cut some concerns post bat cau
Araya, 2021b 12 0.0863223 0.0122300 0.1105895 mobile guided med cbt-based cau cut some concerns follow-up bat cau
Arean, 2016 13 0.2665964 0.0290580 0.1704640 mobile minimal to no support adul not-cbt-based other ctr cut some concerns post pst other ctr
Arjadi, 2018 14 0.3886446 0.0130253 0.1141283 website guided adul cbt-based other ctr mood some concerns post bat other ctr
Baumeister, 2021 15 0.4458137 0.0196181 0.1400648 website guided med cbt-based cau mdd some concerns post cbt cau
Baumeister, 2021 16 0.2640861 0.0193073 0.1389506 website guided med cbt-based cau mdd some concerns follow-up cbt cau
Baumeister, 2021 17 0.2791261 0.0193269 0.1390214 website guided med cbt-based cau mdd some concerns post cbt cau
Baumeister, 2021 18 0.2318410 0.0192687 0.1388118 website guided med cbt-based cau mdd some concerns follow-up cbt cau
Baumeister, 2021 19 0.2138123 0.0192494 0.1387421 website guided med cbt-based cau mdd some concerns post cbt cau
Baumeister, 2021 20 0.1436914 0.0191889 0.1385242 website guided med cbt-based cau mdd some concerns follow-up cbt cau
Baumgartner, 2021 21 0.3175760 0.0089096 0.0943906 website guided adul cbt-based wl cut some concerns post cbt wl
Baumgartner, 2021 22 0.2802350 0.0086312 0.0929041 website guided adul cbt-based wl cut some concerns post cbt wl
Baumgartner, 2021 23 0.3846998 0.0089616 0.0946655 website guided adul cbt-based wl cut some concerns follow-up cbt wl
Baumgartner, 2021 24 0.3772827 0.0086996 0.0932715 website guided adul cbt-based wl cut some concerns follow-up cbt wl
Beevers, 2017 25 0.8144986 0.0153835 0.1240303 website automated encouragement adul cbt-based wl cut high risk post cbt wl
Beevers, 2017 26 0.7632797 0.0152756 0.1235946 website automated encouragement adul cbt-based wl cut high risk post cbt wl
Bendig, 2021 27 0.3154515 0.1195906 0.3458187 website guided med cbt-based wl cut some concerns post cbt wl
Berger, 2011 28 0.6540990 0.0827882 0.2877294 website minimal to no support adul cbt-based wl mood some concerns post cbt wl
Berger, 2011 29 0.1442425 0.0786719 0.2804852 website minimal to no support adul cbt-based wl mood some concerns follow-up cbt wl
Berger, 2011 30 1.1252747 0.0912667 0.3021037 website guided adul cbt-based wl mood some concerns post cbt wl
Berger, 2011 31 0.4470027 0.0804822 0.2836938 website guided adul cbt-based wl mood some concerns follow-up cbt wl
Birney, 2016 32 0.2900000 0.0144000 0.1200000 mobile automated encouragement other group cbt-based other ctr cut some concerns post cbt other ctr
Blanco, 2023 33 1.4800146 0.0884493 0.2974042 mobile guided adul cbt-based other ctr cut low risk post cbt other ctr
Blanco, 2023 34 0.6443551 0.0714296 0.2672631 mobile automated encouragement adul cbt-based other ctr cut low risk post cbt other ctr
Boele, 2018 35 0.6102799 0.0856631 0.2926827 website guided med not-cbt-based wl cut high risk post pst wl
Boeschoten, 2017 36 0.0777749 0.0263541 0.1623395 website guided med not-cbt-based wl cut low risk post pst wl
Braun, 2021 37 0.2806605 0.0118815 0.1090021 website guided other group not-cbt-based other ctr cut some concerns post other psy other ctr
Buntrock, 2015 38 0.6583983 0.0103883 0.1019230 website guided adul not-cbt-based cau cut some concerns post other psy cau
Buntrock, 2015 39 0.2823902 0.0099510 0.0997548 website guided adul not-cbt-based cau cut some concerns follow-up other psy cau
Carlbring, 2013 40 0.8545814 0.0546538 0.2337815 website guided adul cbt-based wl mdd some concerns post 3rd wl
Carlbring, 2013 41 0.6330426 0.0525537 0.2292459 website guided adul cbt-based wl mdd some concerns post 3rd wl
Choi, 2012 42 0.8934932 0.0700034 0.2645816 website guided other group cbt-based wl mdd some concerns post cbt wl
Choi, 2012 43 0.4805137 0.0653867 0.2557081 website guided other group cbt-based wl mdd some concerns post cbt wl
Clarke, 2019 44 -0.0786665 0.0084663 0.0920124 website automated encouragement med cbt-based other ctr cut some concerns post cbt other ctr
Cooper, 2011 45 0.8280493 0.2121586 0.4606068 both minimal to no support med cbt-based cau cut high risk post cbt cau
Cuijpers, 2022a 46 -0.1243773 0.1856208 0.4308373 mobile guided other group cbt-based other ctr cut low risk post bat other ctr
Cuijpers, 2022b 47 0.7325632 0.0186483 0.1365588 mobile guided adul cbt-based other ctr cut low risk post bat other ctr
Dahne, 2019a 48 0.5280632 0.1523608 0.3903342 mobile automated encouragement other group cbt-based cau cut high risk post bat cau
Dahne, 2019a 49 0.4517191 0.2169961 0.4658284 mobile minimal to no support other group cbt-based cau cut high risk post cbt cau
Dahne, 2019b 50 0.3174645 0.1582794 0.3978435 mobile minimal to no support adul cbt-based cau cut high risk post bat cau
Dahne, 2019b 51 0.4274193 0.1702645 0.4126312 mobile minimal to no support adul cbt-based cau cut high risk post cbt cau
Danaher, 2022 52 0.3300421 0.0212304 0.1457066 website automated encouragement ppd cbt-based cau cut some concerns post cbt cau
Deady, 2016 53 0.6229982 0.0753591 0.2745160 website minimal to no support other group cbt-based other ctr cut some concerns post cbt other ctr
Deady, 2016 54 0.2936700 0.1052888 0.3244824 website minimal to no support other group cbt-based other ctr cut some concerns follow-up cbt other ctr
deGraaf, 2009 55 0.1451012 0.0197610 0.1405738 both minimal to no support adul cbt-based cau cut low risk post cbt cau
deGraaf, 2009 56 0.0392583 0.0197126 0.1404014 both minimal to no support adul cbt-based cau cut low risk post cbt NA
deGraaf, 2009 57 0.0976294 0.0197324 0.1404720 both minimal to no support adul cbt-based cau cut low risk follow-up cbt cau
deGraaf, 2009 58 0.1162436 0.0197423 0.1405072 both human encouragement adul cbt-based cau cut low risk follow-up cbt cau
Ebert, 2014 59 0.6905201 0.0282723 0.1681437 website guided other group not-cbt-based wl cut some concerns post pst wl
Ebert, 2014 60 0.3806003 0.0271545 0.1647861 website guided other group not-cbt-based wl cut some concerns follow-up pst wl
Ebert, 2018 61 0.3997118 0.0200024 0.1414297 website human encouragement adul not-cbt-based wl cut some concerns post other psy wl
Ebert, 2018 62 0.3459149 0.0199033 0.1410791 website human encouragement adul not-cbt-based wl cut some concerns post other psy wl
Ebert, 2018 63 0.8383371 0.0213433 0.1460934 website human encouragement adul not-cbt-based wl cut some concerns post other psy wl
Flygare, 2020 64 0.2306238 0.0423944 0.2058991 website guided adul cbt-based other ctr cut some concerns post cbt other ctr
Flygare, 2020 65 0.2635909 0.0424816 0.2061107 website guided adul cbt-based other ctr cut some concerns follow-up cbt other ctr
Flygare, 2020 66 0.1293489 0.0421994 0.2054250 website guided adul cbt-based other ctr cut some concerns follow-up cbt other ctr
Flygare, 2020 67 0.2279166 0.0423878 0.2058830 website guided adul cbt-based other ctr cut some concerns post cbt other ctr
Flygare, 2020 68 0.1761181 0.0422759 0.2056109 website guided adul cbt-based other ctr cut some concerns follow-up cbt other ctr
Flygare, 2020 69 0.0776440 0.0421422 0.2052856 website guided adul cbt-based other ctr cut some concerns follow-up cbt other ctr
Fonseca, 2020 70 -0.0093911 0.0206210 0.1436001 website minimal to no support ppd cbt-based wl cut some concerns post cbt wl
Forand, 2018 71 1.5943651 0.0648136 0.2545851 both guided adul cbt-based wl cut high risk post cbt wl
Forand, 2018 72 1.5537020 0.0763895 0.2763865 both guided adul cbt-based wl cut high risk post cbt wl
Forsell, 2017 73 1.2109857 0.1227672 0.3503816 website guided ppd cbt-based cau mdd some concerns post cbt cau
Forsell, 2017 74 0.5188209 0.1153655 0.3396550 website guided ppd cbt-based cau mdd some concerns post cbt cau
Gaudiano, 2020 75 -0.5802882 0.1046322 0.3234690 website minimal to no support adul cbt-based other ctr mdd high risk post 3rd other ctr
Geraedts, 2014 76 0.2521615 0.0174549 0.1321169 website guided other group cbt-based cau cut some concerns post cbt cau
Ghosh, 2021 77 0.9002485 0.1528561 0.3909681 website minimal to no support adul cbt-based wl cut high risk post cbt wl
Ghosh, 2021 78 0.3480962 0.0530287 0.2302796 website automated encouragement adul cbt-based wl cut high risk post cbt wl
Gilbody, 2015 79 -0.0733015 0.0112938 0.1062721 website human encouragement adul cbt-based cau cut high risk post cbt cau
Gilbody, 2015 80 -0.0876416 0.0105206 0.1025701 website human encouragement adul cbt-based cau cut high risk post cbt cau
Gilbody, 2015 81 0.1048457 0.0163444 0.1278454 website human encouragement adul cbt-based cau cut high risk follow-up cbt cau
Gilbody, 2015 82 0.2295962 0.0163676 0.1279358 website human encouragement adul cbt-based cau cut high risk follow-up cbt cau
Gilbody, 2015 83 -0.0766299 0.0168449 0.1297880 website human encouragement adul cbt-based cau cut high risk follow-up cbt cau
Gilbody, 2015 84 0.1005381 0.0165382 0.1286008 website human encouragement adul cbt-based cau cut high risk follow-up cbt cau
Gili, 2020 85 0.2300000 0.0361000 0.1900000 website automated encouragement adul not-cbt-based cau mood some concerns post other psy cau
Gili, 2020 86 0.3100000 0.0361000 0.1900000 website automated encouragement adul not-cbt-based cau mood some concerns follow-up other psy cau
Gili, 2020 87 0.0100000 0.5184000 0.7200000 website automated encouragement adul not-cbt-based cau mood some concerns follow-up other psy cau
Glozier, 2013 88 0.1618894 0.0083629 0.0914488 website minimal to no support med cbt-based other ctr cut low risk post cbt other ctr
Gräfe, 2020 89 0.2100000 0.0009000 0.0300000 website minimal to no support adul cbt-based cau cut high risk post cbt cau
Gräfe, 2020 90 0.2300000 0.0016000 0.0400000 website minimal to no support adul cbt-based cau cut high risk follow-up cbt cau
Gräfe, 2020 91 0.1500000 0.0016000 0.0400000 website minimal to no support adul cbt-based cau cut high risk follow-up cbt cau
Guo, 2020 92 0.6238601 0.0139853 0.1182594 mobile human encouragement med not-cbt-based wl cut low risk post other psy wl
Guo, 2020 93 0.6012903 0.0139390 0.1180634 mobile human encouragement med not-cbt-based wl cut low risk follow-up other psy wl
Guo, 2020 94 0.5122734 0.0137729 0.1173581 mobile human encouragement med not-cbt-based wl cut low risk follow-up other psy wl
Guo, 2020 95 0.4749678 0.0137112 0.1170949 mobile human encouragement med not-cbt-based wl cut low risk post other psy wl
Guo, 2020 96 0.4979126 0.0137486 0.1172545 mobile human encouragement med not-cbt-based wl cut low risk follow-up other psy wl
Guo, 2020 97 0.3418540 0.0135291 0.1163146 mobile human encouragement med not-cbt-based wl cut low risk follow-up other psy wl
Gupta, 2020 98 -0.2700000 0.0100000 0.1000000 website human encouragement med cbt-based cau cut high risk post cbt cau
Gupta, 2020 99 -0.2800000 0.0121000 0.1100000 website human encouragement med cbt-based cau cut high risk follow-up cbt cau
Gupta, 2020 100 -0.0600000 0.0064000 0.0800000 website human encouragement med cbt-based cau cut high risk post cbt cau
Gupta, 2020 101 -0.1700000 0.0064000 0.0800000 website human encouragement med cbt-based cau cut high risk follow-up cbt cau
Hallgren, 2015 102 0.3206799 0.0082764 0.0909747 website guided adul cbt-based cau cut high risk post cbt cau
Harrer, 2021 103 0.3644587 0.0203346 0.1425995 website automated encouragement young not-cbt-based other ctr cut some concerns post other psy other ctr
Heim, 2021 104 0.7019164 0.0956716 0.3093082 mobile guided adul cbt-based other ctr cut high risk post bat other ctr
Hobfoll, 2016 105 0.2191702 0.0155027 0.1245096 website human encouragement other group cbt-based wl cut some concerns post cbt wl
Hoifodt, 2013 106 0.9235597 0.0418315 0.2045275 website guided adul cbt-based wl cut some concerns post cbt wl
Hoifodt, 2013 107 0.5233640 0.0390602 0.1976365 website guided adul cbt-based wl cut some concerns post cbt wl
Hoifodt, 2013 108 -0.3173972 0.0382314 0.1955286 website guided adul cbt-based wl cut some concerns follow-up cbt wl
Hoifodt, 2013 109 -0.1489189 0.0378554 0.1945647 website guided adul cbt-based wl cut some concerns follow-up cbt wl
Hur, 2018 110 0.6616861 0.1244010 0.3527053 mobile minimal to no support adul cbt-based other ctr mood high risk post cbt other ctr
Jannati, 2020 111 2.9462647 0.1080446 0.3287013 mobile minimal to no support ppd cbt-based wl cut some concerns post cbt wl
Jelinek, 2020 112 0.2786943 0.0722330 0.2687621 website minimal to no support adul cbt-based other ctr cut some concerns post bat other ctr
Jelinek, 2020 113 0.0381744 0.0657450 0.2564079 website minimal to no support adul cbt-based cau cut some concerns post bat cau
Johansson, 2012a 114 1.1111492 0.0503019 0.2242808 website guided adul not-cbt-based wl mdd some concerns post dyn wl
Johansson, 2012a 115 0.9368068 0.0483286 0.2198376 website guided adul not-cbt-based wl mdd some concerns post dyn wl
Johansson, 2012a 116 0.8492545 0.0474643 0.2178631 website guided adul not-cbt-based wl mdd some concerns post dyn wl
Johansson, 2012b 117 0.8261220 0.0580639 0.2409646 website guided adul cbt-based wl mdd some concerns post cbt wl
Johansson, 2012b 118 0.7903957 0.0576709 0.2401476 website guided adul cbt-based wl mdd some concerns post cbt wl
Johansson, 2012b 119 0.5691130 0.0573190 0.2394139 website guided adul cbt-based wl mdd some concerns post cbt wl
Johansson, 2012b 120 0.5590523 0.0572396 0.2392480 website guided adul cbt-based wl mdd some concerns post cbt wl
Johansson, 2019a 121 1.5859228 0.0980525 0.3131334 website guided adul cbt-based wl mdd some concerns post cbt wl
Johansson, 2019a 122 1.5378623 0.0966212 0.3108395 website guided adul cbt-based wl mdd some concerns post cbt wl
Johansson, 2019b 123 0.4356529 0.0284438 0.1686529 website guided med cbt-based wl cut high risk post cbt wl
Johansson, 2019b 124 0.6722977 0.0293639 0.1713590 website guided med cbt-based wl cut high risk post cbt wl
Kivi, 2014 125 0.1570575 0.0658588 0.2566296 website guided adul cbt-based cau mood high risk post cbt cau
Kramer, 2021 126 0.5181935 0.0304165 0.1744033 website guided adul not-cbt-based wl cut some concerns post other psy wl
Kramer, 2021 127 0.5192139 0.0304205 0.1744146 website guided adul not-cbt-based wl cut some concerns post other psy wl
Krämer, 2022 128 1.4360547 0.0208163 0.1442784 website minimal to no support adul not-cbt-based wl mood high risk post other psy wl
Krämer, 2022 129 2.3409554 0.0276935 0.1664136 website minimal to no support adul not-cbt-based wl mood high risk post other psy wl
Krämer, 2022 130 0.9217970 0.0183764 0.1355597 website minimal to no support adul not-cbt-based wl mood high risk post other psy wl
Krämer, 2022 131 2.8290555 0.0953401 0.3087720 website minimal to no support adul not-cbt-based wl mood high risk follow-up other psy wl
Krämer, 2022 132 1.8889909 0.0705534 0.2656189 website minimal to no support adul not-cbt-based wl mood high risk follow-up other psy wl
Krämer, 2022 133 3.2916609 0.7911097 0.8894435 website minimal to no support adul not-cbt-based wl mood high risk follow-up other psy wl
Krämer, 2022 134 1.5792607 0.0216209 0.1470404 website guided adul cbt-based wl mood high risk post cbt wl
Krämer, 2022 135 2.6100029 0.0302747 0.1739962 website guided adul cbt-based wl mood high risk post cbt wl
Krämer, 2022 136 1.2669374 0.0198394 0.1408523 website guided adul cbt-based wl mood high risk post cbt wl
Krämer, 2022 137 2.4263728 0.0822896 0.2868616 website guided adul cbt-based wl mood high risk follow-up cbt wl
Krämer, 2022 138 2.0904062 0.0734957 0.2711009 website guided adul cbt-based wl mood high risk follow-up cbt wl
Krämer, 2022 139 2.1306335 0.3031316 0.5505739 website guided adul cbt-based wl mood high risk follow-up cbt wl
Lambert, 2018 140 0.9166389 0.0886726 0.2977794 website human encouragement adul cbt-based wl cut high risk post bat wl
Lappalainen, 2015 141 0.6247423 0.1109136 0.3330369 website guided adul cbt-based wl mdd some concerns post 3rd wl
Lobner, 2018 142 0.0806035 0.0074494 0.0863101 website minimal to no support adul cbt-based cau mood some concerns post cbt cau
Lobner, 2018 143 0.1726839 0.0085106 0.0922529 website minimal to no support adul cbt-based cau mood some concerns follow-up cbt cau
Lobner, 2018 144 0.0000000 0.0072999 0.0854397 website minimal to no support adul cbt-based cau mood some concerns post cbt cau
Lobner, 2018 145 0.1350230 0.0083799 0.0915416 website minimal to no support adul cbt-based cau mood some concerns follow-up cbt cau
Lokman, 2017 146 0.3867331 0.0123865 0.1112947 website automated encouragement adul cbt-based wl cut some concerns post cbt wl
Lu, 2021 147 0.6648098 0.1217449 0.3489196 website guided adul cbt-based wl cut high risk post cbt wl
Lukas, 2021 148 1.2141225 0.0617941 0.2485841 mobile automated encouragement adul not-cbt-based wl cut high risk post other psy wl
Lundgren, 2016 149 0.2652767 0.0807264 0.2841238 website guided med cbt-based wl cut some concerns post cbt wl
MacKinnon, 2022 150 0.5146510 0.1721647 0.4149274 mobile guided other group not-cbt-based cau cut high risk post other psy cau
MacLean, 2020 151 0.1704545 0.0431745 0.2077849 website guided adul not-cbt-based other ctr cut low risk post other psy other ctr
Meyer, 2015 152 0.5668467 0.0255801 0.1599378 website automated encouragement adul cbt-based wl cut some concerns post cbt wl
Meyer, 2015 153 0.3316252 0.0249257 0.1578789 website automated encouragement adul cbt-based wl cut some concerns follow-up cbt wl
Meyer, 2019 154 0.5369133 0.0207262 0.1439659 website minimal to no support med cbt-based wl mood some concerns post cbt wl
Meyer, 2019 155 0.5076201 0.0206491 0.1436980 website minimal to no support med cbt-based wl mood some concerns post cbt wl
Milgrom, 2016 156 0.8285112 0.1013575 0.3183669 website human encouragement ppd cbt-based cau mood low risk post cbt cau
Milgrom, 2021 157 0.7468071 0.0556522 0.2359071 website human encouragement ppd cbt-based cau mood low risk post cbt cau
Moeini, 2019 158 0.1944934 0.0313995 0.1771992 website guided young cbt-based cau cut some concerns post cbt cau
Moeini, 2019 159 0.2589615 0.0315151 0.1775250 website guided young cbt-based cau cut some concerns follow-up cbt cau
Mohr, 2013 160 0.3964969 0.0600572 0.2450658 website minimal to no support adul cbt-based wl mdd some concerns post cbt wl
Mohr, 2013 161 0.5174917 0.0617604 0.2485164 website human encouragement adul cbt-based wl mdd some concerns post cbt wl
Montero-Marin, 2016 162 0.1208954 0.0325289 0.1803577 website automated encouragement adul cbt-based cau mdd low risk post cbt cau
Montero-Marin, 2016 163 0.3411935 0.0329445 0.1815061 website automated encouragement adul cbt-based cau mdd low risk follow-up cbt cau
Montero-Marin, 2016 164 0.4751333 0.0333908 0.1827316 website automated encouragement adul cbt-based cau mdd low risk follow-up cbt cau
Montero-Marin, 2016 165 0.0773538 0.0305735 0.1748527 website guided adul cbt-based cau mdd low risk post cbt cau
Montero-Marin, 2016 166 0.3820679 0.0311141 0.1763918 website guided adul cbt-based cau mdd low risk follow-up cbt cau
Montero-Marin, 2016 167 0.4832553 0.0314522 0.1773477 website guided adul cbt-based cau mdd low risk follow-up cbt cau
Morgan, 2012 168 0.1644991 0.0070600 0.0840238 website minimal to no support adul not-cbt-based other ctr cut low risk post other psy other ctr
Nadort, 2022 169 0.0616223 0.0332402 0.1823188 website guided med not-cbt-based cau cut some concerns post pst cau
Newby, 2017 170 0.7763902 0.0579911 0.2408134 website guided med cbt-based wl mdd some concerns post cbt wl
Nobis, 2015 171 0.8914534 0.0171873 0.1311003 website guided med not-cbt-based other ctr cut some concerns post other psy other ctr
Nobis, 2015 172 0.8191106 0.0169442 0.1301698 website guided med not-cbt-based other ctr cut some concerns post other psy other ctr
Nyström, 2017 173 0.9727684 0.0388740 0.1971650 website guided adul cbt-based wl mood low risk post bat wl
Nyström, 2017 174 0.6490930 0.0413728 0.2034031 website guided adul cbt-based wl mood low risk post bat wl
O' Moore, 2018 175 1.0077723 0.0702549 0.2650565 website minimal to no support med cbt-based cau mdd some concerns post cbt cau
O' Moore, 2018 176 0.9015138 0.0687512 0.2622045 website minimal to no support med cbt-based cau mdd some concerns follow-up cbt cau
O'Mahen, 2013a 181 0.6472454 0.0594544 0.2438328 website guided ppd cbt-based cau mdd some concerns post bat cau
O'Mahen, 2013a 182 0.4802299 0.0699795 0.2645363 website guided ppd cbt-based cau mdd some concerns follow-up bat cau
O'Mahen, 2013b 183 0.5469195 0.0121357 0.1101620 website automated encouragement ppd cbt-based cau cut some concerns post bat cau
Oehler, 2020 177 0.2328341 0.0152626 0.1235419 website guided adul cbt-based other ctr mood some concerns post cbt other ctr
Oehler, 2020 178 0.1347449 0.0153056 0.1237157 website guided adul cbt-based other ctr mood some concerns post cbt other ctr
Oehler, 2020 179 0.1230026 0.0170787 0.1306854 website guided adul cbt-based other ctr mood some concerns follow-up cbt other ctr
Oehler, 2020 180 0.1113171 0.0170728 0.1306630 website guided adul cbt-based other ctr mood some concerns follow-up cbt other ctr
Perini, 2009 184 0.8469905 0.1008508 0.3175702 website guided adul cbt-based wl mdd high risk post cbt wl
Perini, 2009 185 0.6138469 0.1003006 0.3167026 website guided adul cbt-based wl mdd high risk post cbt wl
Pfeiffer, 2020 186 0.1364048 0.0171114 0.1308107 website guided other group cbt-based cau cut some concerns post cbt cau
Pfeiffer, 2020 187 0.1262654 0.0184122 0.1356916 website guided other group cbt-based cau cut some concerns follow-up cbt cau
Phillips, 2014 188 0.0494895 0.0117830 0.1085495 website human encouragement other group cbt-based other ctr cut some concerns post cbt other ctr
Pots, 2016 189 0.5581645 0.0246195 0.1569059 website guided adul cbt-based wl cut high risk post 3rd wl
Pots, 2016 190 0.3187602 0.0239927 0.1548958 website guided adul cbt-based wl cut high risk follow-up 3rd wl
Pugh, 2016 191 1.0539618 0.1147047 0.3386808 website guided ppd cbt-based wl cut high risk post cbt wl
Pugh, 2016 192 0.8770698 0.1102600 0.3320543 website guided ppd cbt-based wl cut high risk post cbt wl
Raevuori, 2021 193 -0.2581432 0.0325385 0.1803843 mobile guided young not-cbt-based cau mdd some concerns post other psy cau
Raevuori, 2021 194 0.1005334 0.0323077 0.1797435 mobile guided young not-cbt-based cau mdd some concerns follow-up other psy cau
Reins, 2019 195 0.3163818 0.0309227 0.1758484 website guided adul not-cbt-based other ctr mdd some concerns post other psy other ctr
Reins, 2019 196 0.3015156 0.0308872 0.1757475 website guided adul not-cbt-based other ctr mdd some concerns post other psy other ctr
Reins, 2019 197 0.2836912 0.0308469 0.1756329 website guided adul not-cbt-based other ctr mdd some concerns post other psy other ctr
Richards, 2015 198 0.6457757 0.0224044 0.1496808 website guided adul cbt-based wl cut high risk post cbt wl
Richki, 2015a 199 1.1394887 0.1586061 0.3982539 website guided young not-cbt-based wl mdd low risk post other psy wl
Richki, 2015b 200 0.8174530 0.1405297 0.3748729 website guided young not-cbt-based wl mdd some concerns post other psy wl
Ritvo, 2021 201 0.4429332 0.0911912 0.3019789 website guided young cbt-based cau mdd high risk post 3rd cau
Ritvo, 2021 202 0.7014920 0.0945975 0.3075671 website guided young cbt-based cau mdd high risk post 3rd cau
Ritvo, 2021 203 0.6947830 0.0944897 0.3073917 website guided young cbt-based cau mdd high risk post 3rd cau
Roepke, 2015 204 0.3180025 0.0787064 0.2805466 website minimal to no support adul cbt-based wl cut high risk post cbt wl
Roepke, 2015 205 0.7772986 0.0890935 0.2984853 website minimal to no support adul cbt-based wl cut high risk post 3rd wl
Rosso, 2017 206 0.7856270 0.0561165 0.2368893 website human encouragement adul cbt-based other ctr mdd low risk post cbt other ctr
Rosso, 2017 207 0.7811842 0.0560704 0.2367919 website human encouragement adul cbt-based other ctr mdd low risk post cbt other ctr
Ruehlman, 2021 208 0.7721817 0.0832911 0.2886020 website minimal to no support young cbt-based wl cut high risk post cbt wl
Ruwaard, 2009 209 0.8473307 0.0901782 0.3002968 website guided adul cbt-based wl cut high risk post cbt wl
Ruwaard, 2009 210 0.8309269 0.0899157 0.2998595 website guided adul cbt-based wl cut high risk post cbt wl
Ruwaard, 2009 211 0.5196580 0.0859078 0.2931004 website guided adul cbt-based wl cut high risk post cbt wl
Salamanca-Sanabria, 2020 212 0.8979870 0.0716260 0.2676304 website guided young cbt-based wl cut high risk post cbt wl
Sander, 2020 215 0.4228443 0.0138653 0.1177511 website guided med cbt-based cau cut some concerns post cbt cau
Sander, 2020 216 0.2753403 0.0136899 0.1170038 website guided med cbt-based cau cut some concerns follow-up cbt cau
Sander, 2020 217 0.4021581 0.0138363 0.1176276 website guided med cbt-based cau cut some concerns follow-up cbt cau
Sander, 2020 218 0.3284546 0.0137445 0.1172370 website guided med cbt-based cau cut some concerns follow-up cbt cau
Sander, 2020 219 0.3429331 0.0137611 0.1173076 website guided med cbt-based cau cut some concerns follow-up cbt cau
Sawyer, 2019 262 0.2080483 0.0362601 0.1904208 mobile guided ppd cbt-based cau cut low risk post cbt cau
Sawyer, 2019 263 -0.2496580 0.0363471 0.1906490 mobile guided ppd cbt-based cau cut low risk follow-up cbt cau
Schleider, 2022 213 0.1969190 0.0024644 0.0496428 website minimal to no support young not-cbt-based other ctr cut some concerns post other psy other ctr
Schleider, 2022 214 0.2196713 0.0024553 0.0495505 website minimal to no support young cbt-based other ctr cut some concerns post bat other ctr
Schlicker, 2020 220 0.4255328 0.0539937 0.2323654 website guided med cbt-based wl mdd some concerns post cbt wl
Schlicker, 2020 221 0.2972396 0.0533710 0.2310217 website guided med cbt-based wl mdd some concerns post cbt wl
Schlicker, 2020 222 0.2466567 0.0531863 0.2306216 website guided med cbt-based wl mdd some concerns follow-up cbt wl
Schlicker, 2020 223 0.3714769 0.0537044 0.2317421 website guided med cbt-based wl mdd some concerns follow-up cbt wl
Schure, 2019 224 0.5255538 0.0121021 0.1100096 website automated encouragement adul cbt-based wl cut some concerns post cbt wl
Seo, 2022 225 0.4222655 0.0560524 0.2367539 mobile human encouragement ppd cbt-based other ctr cut high risk post cbt other ctr
Sheeber, 2012 226 0.8402299 0.0632160 0.2514279 website guided ppd cbt-based wl cut some concerns post cbt wl
Sheeber, 2017 227 0.3133597 0.0152241 0.1233859 website guided ppd cbt-based other ctr cut some concerns post cbt other ctr
Sheeber, 2017 228 0.3083201 0.0152182 0.1233619 website guided ppd cbt-based other ctr cut some concerns post cbt other ctr
Smith, 2017 229 0.8550238 0.0557363 0.2360854 website human encouragement adul cbt-based wl mdd high risk post cbt wl
Spek, 2007 230 0.2667666 0.0199814 0.1413556 website minimal to no support adul cbt-based wl cut some concerns post cbt wl
Stiles-Shields, 2019 231 0.9311597 0.2236313 0.4728967 mobile human encouragement adul cbt-based wl cut high risk post bat wl
Stiles-Shields, 2019 232 1.5085550 0.3171542 0.5631644 mobile human encouragement adul cbt-based wl cut high risk post cbt wl
Stuart, 2022 233 0.5563872 0.0239751 0.1548390 website automated encouragement adul cbt-based cau cut some concerns post cbt cau
Stuart, 2022 234 0.1815184 0.0210518 0.1450922 website automated encouragement adul cbt-based cau cut some concerns follow-up cbt cau
Sun, 2021 235 0.4809143 0.0353930 0.1881301 mobile minimal to no support ppd cbt-based other ctr cut some concerns post 3rd other ctr
Sun, 2021 236 -0.1058622 0.0442927 0.2104583 mobile minimal to no support ppd cbt-based other ctr cut some concerns follow-up 3rd other ctr
Titov, 2010 237 1.0784313 0.0567081 0.2381346 website human encouragement adul cbt-based wl mdd some concerns post cbt wl
Titov, 2010 238 1.0730864 0.0535555 0.2314207 website guided adul cbt-based wl mdd some concerns post cbt wl
Titov, 2010 239 1.2584710 0.0593554 0.2436296 website human encouragement adul cbt-based wl mdd some concerns post cbt wl
Titov, 2010 240 1.2617210 0.0561626 0.2369865 website guided adul cbt-based wl mdd some concerns post cbt wl
Unlu-Ince, 2013 241 0.7091438 0.0764129 0.2764288 website guided other group not-cbt-based wl cut some concerns post pst wl
van Bastelaar, 2011 242 0.4915783 0.0255090 0.1597153 website guided med cbt-based wl cut high risk post cbt wl
Van Luenen, 2018 243 0.6089118 0.0222924 0.1493064 website guided med cbt-based wl cut low risk post cbt wl
Van Luenen, 2018 244 0.7635636 0.0228615 0.1512002 website guided med cbt-based wl cut low risk post cbt wl
Vernmark, 2010 245 0.9293583 0.0766157 0.2767954 both guided adul cbt-based wl mdd some concerns post cbt wl
Vernmark, 2010 246 0.8008440 0.0746462 0.2732146 both guided adul cbt-based wl mdd some concerns post cbt wl
Vernmark, 2010 247 0.5566486 0.0743652 0.2726999 website guided adul cbt-based wl mdd some concerns post cbt wl
Vernmark, 2010 248 0.3559166 0.0726831 0.2695980 website guided adul cbt-based wl mdd some concerns post cbt wl
Vigod, 2021 249 0.1483359 0.0521728 0.2284137 website guided ppd not-cbt-based wl cut high risk post ipt wl
Warmerdam, 2008 250 0.5434037 0.0237089 0.1539771 website guided adul cbt-based wl cut some concerns post cbt wl
Warmerdam, 2008 251 0.4309754 0.0233932 0.1529484 website guided adul not-cbt-based wl cut some concerns post pst wl
Weise, 2019 252 0.9709331 0.0296132 0.1720847 website guided adul cbt-based wl mood high risk post cbt wl
Westerhof, 2019 253 0.2230608 0.1613205 0.4016473 website guided adul not-cbt-based wl cut some concerns post lrt wl
Westerhof, 2019 254 0.7078528 0.1859672 0.4312391 website guided adul not-cbt-based wl cut some concerns follow-up lrt wl
Williams, 2013a 255 0.9394896 0.1063702 0.3261444 website guided adul cbt-based wl mdd high risk post cbt wl
Williams, 2013a 256 0.9355157 0.1062781 0.3260031 website guided adul cbt-based wl mdd high risk post cbt wl
Williams, 2022 257 0.1274222 0.1116910 0.3342020 website guided adul cbt-based wl cut high risk post cbt wl
Ying, 2022 258 3.2995380 0.0432942 0.2080725 website guided adul cbt-based wl cut high risk post cbt wl
Ying, 2022 259 1.5625309 0.0238782 0.1545257 website guided adul cbt-based wl cut high risk post cbt wl
Ying, 2022 260 2.5786107 0.0335517 0.1831713 website guided adul cbt-based wl cut high risk post cbt wl
Zhao, 2022 261 1.2690885 0.0264824 0.1627342 website minimal to no support young cbt-based wl cut some concerns post 3rd wl

Step 1: Data Validation

Before proceeding with a multiverse meta-analysis, it’s important to ensure your data meets the requirements:

Code 
# Check if data meets requirements for multiverse analysis
check_data_multiverse(data_digDep)
Data check passed. Dataset is valid.
[1] TRUE
What the validator checks

The check_data_multiverse() function validates:

  • Required columns are present (study, es_id, yi, vi, and wf_* columns)
  • Columns have correct data types
  • Effect size IDs are unique
  • No critical missing values

Step 2: Create Specifications

Principled Selection of Which and How Factors

When conducting a multiverse meta-analysis, a critical step is determining which analytic choices (decision nodes) should be included in your multiverse. Not all potential choices merit inclusion, and an overly expansive multiverse can obscure meaningful effects with unjustified alternatives. Following Del Giudice and Gangestad (2021), we recommend a systematic approach to evaluating decision nodes.

Three Types of Nonequivalence to Consider

Before including factors in your multiverse, assess them against these criteria:

  1. Measurement Nonequivalence: Do different measurement choices have different validities or reliabilities?
    • Individual indicators vs. composites
    • Validated vs. non-validated measures
    • Measures with different psychometric properties
  2. Effect Nonequivalence: Do different specifications capture different effects?
    • Different covariate sets that change the nature of the estimated effect
    • Direct vs. total effects in mediation scenarios
    • Incompatible causal assumptions
  3. Power/Precision Nonequivalence: Do alternatives yield predictably different levels of precision?
    • Sample size differences due to exclusion criteria
    • Reliability differences affecting statistical power
    • Measurement approaches with different sensitivities

Decision Framework for Each Factor

For each decision node in your analysis, classify it into one of these three categories:

  1. Type E (Principled Equivalence): Alternatives that are objectively equivalent
    • Choices with comparable validity and reliability
    • Options that test the same effect with similar precision
    • These can be included in a single multiverse
  2. Type N (Principled Nonequivalence): Alternatives where some are objectively better
    • One measurement approach has clearly higher validity
    • Some specifications reflect bias in effect estimation
    • These should NOT be combined in a single multiverse
    • Choose the theoretically justified option(s)
  3. Type U (Uncertainty): Cases where optimal choice is unclear
    • Insufficient evidence to determine superior options
    • Genuine theoretical uncertainty about causal relationships
    • Create separate multiverses for conceptually distinct models

Step-by-Step Evaluation Process

  1. Identify all potential decision nodes in your analysis:
    • Which predictors or outcomes to use
    • How to operationalize key constructs
    • Which covariates to include
    • Data exclusion criteria
  2. Evaluate each decision node using the three nonequivalence criteria:
    • Research the psychometric properties of measures
    • Consider causal relationships between variables
    • Assess impact on statistical power and precision
  3. Classify each decision as Type E, N, or U:
    • Document your reasoning for each classification
    • For Type U decisions, specify what information would resolve uncertainty
  4. Construct appropriate multiverses:
    • Include Type E decisions in a single homogeneous multiverse
    • For Type N decisions, select justified options only
    • Create separate multiverses for Type U decisions

This approach leads to more interpretable and theoretically meaningful results by avoiding the creation of multiverse “black holes” where true effects are obscured by a sea of unjustified alternatives.

Example Application

For our depression and inflammation example:

  • Choice of composites (Type E): Using a composite of all validated depression instruments, or excluding potentially problematic ones with minimal impact
  • Outlier criteria (Type E): Different reasonable thresholds when sample size impact is minimal
  • Including fatigue as a covariate (Type U): Create two separate multiverses, one treating fatigue as a collider, another as a mediator
  • Including unvalidated biomarkers (Type N): Exclude from the multiverse analyses
  • Including inadequate meta-analytical models (Type N): Models that are arguably indefensible, for instance using a common effect model where no theoretical argument can be made for only varying around a fixed effect, or using an unweighted “voting model”, where each study gets one vote.
  • Including unvalidated biomarkers (Type N): Exclude from the multiverse analyses

By applying this framework, we ensure our multiverse analysis is both comprehensive where appropriate and focused where theory provides clear guidance.

Next, we define the specifications to explore in our multiverse analysis:

Code 
# Define "Which factors" (analytical choices about data subsets)
wf_vars <- c("wf_1", "wf_2", "wf_3", "wf_4")

# Define meta-analytical methods
ma_methods <- c("reml", "fe", "3-level", "rve", "p-uniform")

# Define dependency handling approaches
dependencies <- c("aggregate", "modeled")

# Create specifications grid
specs <- create_multiverse_specifications(
  data = data_digDep,
  wf_vars = wf_vars,
  ma_methods = ma_methods,
  dependencies = dependencies
)

# Display number of specifications
tibble(
  "Description" = "Number of valid specifications",
  "Value" = specs$number_specs
) %>%
  kable(format = "html") %>%
  kable_styling(full_width = FALSE, position = "left")
Description Value
Number of valid specifications 1800
Code 
# Look at first few specifications
specs$specifications %>%
  head(10) %>% 
  kable(format = "html") %>%
  kable_styling(
    bootstrap_options = c("striped", "hover", "condensed"),
    full_width = FALSE
  ) %>%
  column_spec(1:ncol(specs$specifications), 
              background = if_else(
                grepl("total_", as.matrix(specs$specifications[1:10,])), 
                "#e8f4f8", "white"
              ))
Warning in ensure_len_html(background, nrows, "background"): The number of
provided values in background does not equal to the number of rows.
wf_1 wf_2 wf_3 wf_4 dependency ma_method row_id
website minimal to no support adul not-cbt-based aggregate reml 1
mobile minimal to no support adul not-cbt-based aggregate reml 2
both minimal to no support adul not-cbt-based aggregate reml 3
total_wf_1 minimal to no support adul not-cbt-based aggregate reml 4
website guided adul not-cbt-based aggregate reml 5
mobile guided adul not-cbt-based aggregate reml 6
both guided adul not-cbt-based aggregate reml 7
total_wf_1 guided adul not-cbt-based aggregate reml 8
website automated encouragement adul not-cbt-based aggregate reml 9
mobile automated encouragement adul not-cbt-based aggregate reml 10
Understanding specifications

Each row in the specifications represents a unique analytical path through the multiverse. The columns indicate:

  • Which data subsets to use (which factors, wf_* columns)
  • How to handle dependencies between effect sizes (dependency)
  • Which meta-analytical method to apply (ma_method)

Step 3: Run Multiverse Analysis

With our specifications defined, we can now run the multiverse analysis:

Code 
# Run the multiverse analysis
results <- run_multiverse_analysis(
  data_multiverse = data_digDep,
  specifications = specs$specifications,
  how_methods = ma_methods
)

# View summary of results
summary_stats <- results %>%
  summarize(
    n_analyses = n(),
    mean_es = mean(b, na.rm = TRUE),
    median_es = median(b, na.rm = TRUE),
    min_es = min(b, na.rm = TRUE),
    max_es = max(b, na.rm = TRUE),
    prop_significant = mean(pval < 0.05, na.rm = TRUE)
  )

summary_stats %>%
  tidyr::pivot_longer(cols = everything(), names_to = "Statistic", values_to = "Value") %>%
  mutate(
    Value = case_when(
      Statistic == "prop_significant" ~ paste0(round(Value * 100, 1), "%"),
      TRUE ~ round(Value, 3) %>% as.character()
    ),
    Statistic = case_when(
      Statistic == "n_analyses" ~ "Number of Analyses",
      Statistic == "mean_es" ~ "Mean Effect Size",
      Statistic == "median_es" ~ "Median Effect Size",
      Statistic == "min_es" ~ "Minimum Effect Size",
      Statistic == "max_es" ~ "Maximum Effect Size",
      Statistic == "prop_significant" ~ "Proportion Significant (p < 0.05)",
      TRUE ~ Statistic
    )
  ) %>%
  kable(format = "html", caption = "Summary of Multiverse Meta-Analysis Results") %>%
  kable_styling(
    bootstrap_options = c("striped", "hover"),
    full_width = FALSE,
    position = "center"
  ) %>%
  row_spec(row = 0, bold = TRUE, color = "white", background = "#2c3e50")
Summary of Multiverse Meta-Analysis Results
Statistic Value
Number of Analyses 342
Mean Effect Size 0.48
Median Effect Size 0.468
Minimum Effect Size 0.028
Maximum Effect Size 2.791
Proportion Significant (p < 0.05) 91.2%

Step 4: Visualize Results

The metaMultiverse package provides several visualization options to explore and interpret your results.

Specification Curve

The specification curve visualizes the distribution of effect sizes across all specifications:

Code 
# Define a lookup table for factor labels
factor_label_lookup <- list(
  wf_1 = "Technology Type",
  wf_2 = "Guidance Level",
  wf_3 = "Target Group",
  wf_4 = "Therapy Type",
  ma_method = "Meta-Analysis Method"
)

# Create the specification curve plot
plotly_descriptive_spec_curve(
  data = results,
  ylim_lower = -0.5,
  ylim_upper = 1.5,
  colorblind_friendly = TRUE,
  factor_label_lookup = factor_label_lookup
)
Figure 1: Specification curve showing the distribution of effect sizes across all analytical choices.

The specification curve consists of two panels:

  1. Top panel: Shows the effect size estimates with confidence intervals, ordered by effect size magnitude
  2. Bottom panel: Shows which analytical choices were made for each specification

Vibration of Effects Plot

The Vibration of Effects plot shows the relationship between effect sizes and p-values:

Code 
plotly_VoE(
  data = results,
  x = "b",
  y = "pval",
  colorblind_friendly = TRUE,
  cutoff = 5,  # Minimum number of studies
  vertical_lines = c(0.1, 0.9),  # Quantile reference lines
  hline_value = 0.05  # Significance threshold
)
Figure 2: Vibration of Effects plot showing relationship between effect sizes and p-values.
Interpreting the Vibration of Effects plot
  • Points represent individual meta-analyses
  • Color intensity represents point density
  • Horizontal line: conventional significance threshold (p = 0.05)
  • Vertical lines: effect size distribution quantiles (10th and 90th percentiles)

Interpreting Multiverse Results

When interpreting results from a multiverse meta-analysis, consider:

Consistency of direction: Do most specifications suggest effects in the same direction?

A finding is more robust if the majority of specifications point to effects in the same direction, even if magnitude varies.

Range of effect sizes: How much do effect sizes vary across specifications?

A narrow range suggests conclusions are stable across analytical choices, while wide ranges indicate high sensitivity to analytical decisions.

Analytical decisions driving variation: Which analytical choices have the largest impact on results?

Identify which factors (e.g., inclusion criteria, meta-analytical method) most strongly influence effect size estimates.

Proportion of significant results: What proportion of specifications yield statistically significant results?

A high proportion of significant results across the multiverse suggests a robust finding.

Decision sensitivity: Are conclusions robust to different analytical choices?

Assess whether key conclusions change substantively depending on analytical choices.

Advanced Usage

Custom Meta-Analytical Methods

You can extend the package by adding custom meta-analytical methods:

Code 
# Example of defining a custom meta-analytical function
calculate_custom_method <- function(dat) {
  # Your custom implementation here
  
  # Return results in standardized format
  list(
    b = estimate,
    ci.lb = lower_ci,
    ci.ub = upper_ci,
    pval = p_value
  )
}

# Add to how_methods
custom_methods <- c(ma_methods, "custom_method")

Subset Analysis

You can focus on a subset of specifications for targeted analysis:

Code 
# Focus on specifications with specific characteristics
subset_specs <- specs$specifications %>%
  filter(
    wf_1 == "specific_value",
    dependency == "modeled"
  )

# Run analysis on subset
#subset_results <- run_multiverse_analysis(
#  data_multiverse = data_digDep,
#  specifications = subset_specs,
#  how_methods = ma_methods
#)

Conclusion

The metaMultiverse package provides a systematic framework for exploring the impact of analytical decisions in meta-analysis. By conducting and visualizing multiple analyses simultaneously, researchers can:

  1. Identify which analytical choices significantly impact conclusions
  2. Present more transparent and robust meta-analytic results
  3. Better understand the sensitivity of findings to methodological decisions
Best practices for multiverse meta-analysis
  • Pre-register your multiverse analysis plan when possible
  • Clearly document all analytical choices
  • Report the full range of results rather than cherry-picking
  • Consider theoretical justifications for different analytical choices
  • Use visualizations to communicate the multiverse of results effectively

For more information, see the package documentation or contact the developers.